{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contigency Tables for McNemar's Test and Cochran's Q Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function to compute a 2x2 contingency tables for McNemar's Test and Cochran's Q Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> `from mlxtend.evaluate import mcnemar_tables`    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Contigency Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A 2x2 contigency table as being used in a McNemar's Test ([`mlxtend.evaluate.mcnemar`](mcnemar.md)) is a useful aid for comparing two different models. In contrast to a typical confusion matrix, this table compares two models to each other rather than showing the false positives, true positives, false negatives, and true negatives of a single model's predictions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./mcnemar_table_files/mcnemar_contingency_table.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, given that 2 models have a accuracy of with a 99.7% and 99.6% a 2x2 contigency table can provide further insights for model selection.\n",
    "\n",
    "\n",
    "![](./mcnemar_table_files/mcnemar_contingency_table_ex1.png)\n",
    "\n",
    "In both subfigure A and B, the predictive accuracies of the two models are as follows:\n",
    "\n",
    "- model 1 accuracy: 9,960 / 10,000 = 99.6%\n",
    "- model 2 accuracy: 9,970 / 10,000 = 99.7%\n",
    "\n",
    "Now, in subfigure A, we can see that model 2 got 11 predictions right that model 1 got wrong. Vice versa, model 2 got 1 prediction right that model 2 got wrong. Thus, based on this 11:1 ratio, we may conclude that model 2 performs substantially better than model 1. However, in subfigure B, the ratio is 25:15, which is less conclusive about which model is the better one to choose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "- McNemar, Quinn, 1947. \"[Note on the sampling error of the difference between correlated proportions or percentages](http://link.springer.com/article/10.1007%2FBF02295996)\". Psychometrika. 12 (2): 153–157.\n",
    "- Edwards AL: Note on the “correction for continuity” in testing the significance of the difference between correlated proportions. Psychometrika. 1948, 13 (3): 185-187. 10.1007/BF02289261.\n",
    "- https://en.wikipedia.org/wiki/McNemar%27s_test\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1 - Single 2x2 Contigency Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_0 vs model_1': array([[ 4.,  1.],\n",
       "        [ 2.,  3.]])}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from mlxtend.evaluate import mcnemar_tables\n",
    "\n",
    "y_true = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])\n",
    "\n",
    "y_mod0 = np.array([0, 1, 0, 0, 0, 1, 1, 0, 0, 0])\n",
    "y_mod1 = np.array([0, 0, 1, 1, 0, 1, 1, 0, 0, 0])\n",
    "\n",
    "tb = mcnemar_tables(y_true, \n",
    "                    y_mod0, \n",
    "                    y_mod1)\n",
    "\n",
    "tb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize (and better interpret) the contigency table via matplotlib, we can use the `checkerboard_plot` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAREAAADrCAYAAABO6qJFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAADc1JREFUeJzt3X1sXfV9x/H3hzgyMEK3MuqsoBKkddDxoKhNmQh0ydZuApWpBY1KUwOKRLZpUlVaCTRYNi1dmYI2tYtUbaP0CRhdi9gkMkZAkKzJlvIQHpqKICADtUFkJCn/jLU0iRO++8MnrUttJ/hn514775d05eNzzj3n6yP7fY9voiRVhSRN1nG9HkDSzGZEJDUxIpKaGBFJTYyIpCZGRFITIyKpiRGR1MSISGpiRCQ1MSKSmhgRSU2MiKQmRkRSEyMiqYkRkdTEiEhqYkQkNTEikpoYEUlNjIikJkZEUhMjMgslWZ9kwQTbVyVZ9qZ1pyfZmOS/knw7yaLpnnMmOlrXNsmaJKdOsH15kj8fY/1Hk7zrcMefSgNH82Tqa/8HfKyq9iT5deCLwAcmc6Akc6rq4JRON7O9pWvbXb9PTfJcHwVeBV6a5PPfMu9E+kiSBUm2JPlakm1JPp7k9iRPJbmx22coyf1JNiVZd+jVKsm1SZ5I8nXgbaOOubrb95Ekl4137qr636ra0326HzgwxnxPJjkuye8leaVbd2WSP+tmfzzJPwFfmmDOF5J8plt/V7duTpJ/7tbdnOSFqbmiPzN7v1/bVUluS/JvwMe6O5fTD3Ntzkvyr93X84EuUJcAX0hyd/NFO1JV5aNPHsACYCdwPDAf2Nt9HABe7PZZA1zdLV8NfB54B/AdYC5wMiOvRAu6b6hbun1PBL4LBFgFLBtnhjnAA8CHxtj2FeB9wN8Ba4FzgH8ELuzO9wPg5PHm7Ja/Dyzslh8EzgWuGDXnRcD3j8Fruwr44qjPNwKnj3dtgOXAPd3yYuBfuuXbgIuP5vetdyL957mq2ltVu4CdVbWrqg4AP04yBzgLeLjb92HgbOBMYFtVDVfVa8Bz3fbzgCVJNgLrgEHglMOc/4vAfVW1foxtG4APAr8G/H23vAh4vNu+rTs/48wJcKCqtnbLL3XzvHvUMR4Dpuv/du3na3vonG820bV5svt46Dr2hBHpPzXOMoy80j3PyCsP3cfnge8B5yQZSDKPn/7APgM8WFVLq2opcH5VvTreiZP8LfBKVX1hnF3+A/h9Rl6NNwNXAnu6H0SA0e+DjDXnmKcFXmAkRgDv79ZNh36+tvCz1++Qia7N6K/h0Pr9HOX3On1jdea5Gbg9yQrgdUZuv/ckuZORV6rtjHzjU1XrklzYvVoW8DJw1VgH7f7E4FPAt7v9f1BVV47ep6p2JfkFYGNVvZ7kDeBbRzrnBF/TPcCVSTYx8qq773AXYZr07NpO4K1em38H/irJs1X1x0d4jibpfo+SeirJ3KoaTnIRcGNVjftG5bGm36+NdyLqF99M8suMvLdwVF5BZ5C+vjbeiUhq4hurkpoYEUlNjIikJkZEUhMjIqmJEZHUxIhIamJEJDUxIpKaGBFJTYyIpCZGRFITIyKpiRGR1MSISGriP0p0FJ1wwgm79u7dO9TrOWaRN/CFcKrsrqr5k3mi/yjRUZSkvN5TJ5muf8/52FRVk7qgVlxSEyMiqYkRkdTEiEhqYkQkNTEikpoYEUlNjIikJkZEUhMjIqmJEZHUxIhIamJEJDUxIpKaGBFJTYyIpCZGRFITIyKpiRGR1MSISGpiRCQ1MSKSmhgRSU2MiKQmRkRSEyMiqYkRkdTEiEhqYkQkNTEikpoYEUlNjIikJkZEUhMjIqlJX0QkyfokCybYvirJsjHWfzXJK0m+PJ3zHUu2b9/O3Llz2bx5c69HmfEeeOAB9uzZw8qVK3s9yrTqi4g0+AvgD1oPkmTOFMwyK3z2s59lyZIlvR5jVrjmmmu4/vrrez3GtGuKSJIFSbYk+VqSbUk+nuT2JE8lubHbZyjJ/Uk2JVmX5NRu/bVJnkjydeBto465utv3kSSXTXT+qtp5mPk2JHl7kvOS7E8yL8n7k9zabd+R5B+AtUlOTHJ3d+5vJfnVbp+NSW5O8mB3vMFu/ee6GW9JsqPlOvaLLVu2MH/+fE4//fRejzIr7Nw54bfnrDEVdyKnAX8CfAj4CvCnwAXAim77jcA3qmoJ8E3gxiTvAJYDF3bPPRMgySXAL3X7fhD46yRpmG0j8FvAbwP3A7/ZLX+r2/4rwM1VdRnwR8DT3bn/EvibUcd5tKp+F3gR+J0k7wXOqaoLgdXAOxtm7Bs33XQTN9xwQ6/H0AwzFRF5rqr2VtUuYGdV7aqqA8CPu18TzgIe7vZ9GDibkWhsq6rhqnoNeK7bfh6wJMlGYB0wCJzSMNsGRmJ0EfCZbnkpP43Izqp6qVsea85Dnuw+vtTN827gcYCq2gHsbpixL9x3330sWrSIU05pudw6Fk1FRGqcZYAAzwOLu88Xd59/DzgnyUCSefz0B/YZ4MGqWlpVS4Hzq+rVhtm2AL8BDFbVU8A5wCld8AAOjtp3rDnH+roCvAC8DyDJu4Chhhn7wtatW9m4cSOXXHIJDz30ENdddx07dsyK39I0zQaOwjluBm5PsgJ4Hbi6qvYkuRN4DNjOSFSoqnVJLuzuRAp4GbhqvAMnuQm4FJifZD3wkar60aHtVXUgyS5ga7dqF/Df4xzuS8AdSf6zO/cfjnfeqnoyyfYkjwDbgBn/y+/KlSt/8qcIy5cvZ8WKFZxxxhk9nmpmu/XWW1m8eDGDg4MsWrSIyy+/vNcjTYtUvfnmQUciydyqGk5yBrC2qhYewXPK6z112t4u05tV1aQu6NG4E5mt1iQ5FzgJuK7Xw0i94p3IUeSdyNTyTmRqTfZOZKb/ZTNJPWZEJDUxIpKaGBFJTYyIpCZGRFITIyKpiRGR1MSISGpiRCQ1MSKSmhgRSU2MiKQmRkRSEyMiqYkRkdTEiEhqYkQkNTEikpoYEUlNjIikJkZEUhMjIqmJEZHUxIhIamJEJDUxIpKaGBFJTYyIpCZGRFITIyKpiRGR1MSISGpiRCQ1MSKSmgz0eoBjzBtJDPcUGRwcZN++fb0eY1YYHBx8Y7LPTVVN5SyaQBIv9hTz+3dqJKGqMpnn+qooqYkRkdTEiEhqYkQkNTEikpoYEUlNjIikJkZEUhMjIqmJEZHUxIhIamJEJDUxIpKaGBFJTYyIpCZGRFITIyKpiRGR1MSISGpiRCQ1MSKSmhgRSU2MiKQmRkRSEyMiqYkRkdTEiEhqYkQkNTEikpoYEUlNjIikJkZEUhMjIqmJEZHUxIhIatLXEUmyPsmCCbavSrJsjPVfTfJKki8f4XluSHLeBNuXjnWsbv35R3KOmWDhwoVs3ryZTZs2sWHDBs4888xejzRjvfbaayxevJilS5dywQUXsGHDhl6PNH2qqm8fwHpgwQTbVwHLxlh/GrAU+PIRnGPOEewz5rHGO/8Ex6l+fgwNDdVJJ51UQF166aV1xx139Hymwz361cGDB2t4eLiqql588cVatGhRjyeaWHctJ/VzOi13IkkWJNmS5GtJtiX5eJLbkzyV5MZun6Ek9yfZlGRdklO79dcmeSLJ14G3jTrm6m7fR5JcNtH5q2rnYeZbnuTuJPcA1ya5LcnF3bbPdee4JcmOUU87Lck3kjyd5MokbweWAyuTbEwyZzLXqp/s3r2bH/7whwDs37+fAwcO9Hiimeu4445jYGAAGLkrOf/8WXPD+vMmW5+JHsACYCdwPDAf2Nt9HABe7PZZA1zdLV8NfB54B/AdYC5wMvBqd6xLgFu6fU8EvguECe4EmOBOhJEf/geAdJ/fBlwMvBd4oFt3BjA86lhPAnOAdwJP1Cy8Ezn0OPHEE+vRRx+t97znPT2f5XCPfvbyyy/XRRddVKeeemrde++9vR5nQvTbnUjnuaraW1W7gJ1VtauqDgA/7l61zwIe7vZ9GDgbOBPYVlXDVfUa8Fy3/TxgSZKNwDpgEDilcb5Huos32ruBxwGqagewe9S2rVV1sKr+B/jFxnP3rYGBAe666y5Wr17Ns88+2+txZrTTTjuNzZs3s2XLFj7xiU/0epxpM50RqXGWYeQu4nlgcff54u7z7wHnJBlIMo+RsAA8AzxYVUurailwflW92jjfwTHWvQC8DyDJu4ChCb4GgP2M3F3NCkm48847ueeee1i7dm2vx5nR9u3b95Plk08+mXnz5vVwmunVyx+Am4Hbk6wAXmfkV5s9Se4EHgO2MxIVqmpdkgu7O5ECXgauGu/ASW4CLgXmJ1kPfKSqfnS4garqySTbkzwCbGPkV7KJPASs6d6j+VhVvXG4c/SzK664gg9/+MMMDQ2xbNkynn76aT75yU/2eqwZadu2bXz6059mzpw5DA8Ps2bNml6PNG3y83f0x7Ykc6tqOMkZwNqqWjiFx/ZiTzG/f6dGEqoqk3nurLkVn0JrkpwLnARc1+thpH7nnchR5J3I1PP7d2q03In09d9YldT/jIikJkZEUhMjIqmJEZHUxIhIamJEJDUxIpKaGBFJTYyIpCZGRFITIyKpiRGR1MSISGpiRCQ1MSKSmhgRSU2MiKQmRkRSEyMiqYkRkdTEiEhqYkQkNTEikpoYEUlNjIikJkZEUhMjIqmJEZHUxIhIamJEJDUxIpKaGBFJTYyIpCZGRFKTgV4PcIzZDQz1eojZYnBw8I0kvhBOgeOPP373ZJ+bqprKWSQdY6y4pCZGRFITIyKpiRGR1MSISGpiRCQ1MSKSmhgRSU2MiKQmRkRSEyMiqYkRkdTEiEhqYkQkNTEikpoYEUlNjIikJkZEUhMjIqmJEZHUxIhIamJEJDX5f1Nk5fWRxpQGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10c56acc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mlxtend.plotting import checkerboard_plot\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "brd = checkerboard_plot(tb['model_0 vs model_1'],\n",
    "                        figsize=(3, 3),\n",
    "                        fmt='%d',\n",
    "                        col_labels=['model 2 wrong', 'model 2 right'],\n",
    "                        row_labels=['model 1 wrong', 'model 1 right'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2 - Multiple 2x2 Contigency Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If more than two models are provided as input to the `mcnemar_tables` function, a 2x2 contingency table will be created for each pair of models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_0 vs model_1 \n",
      " [[ 4.  1.]\n",
      " [ 2.  3.]] \n",
      "\n",
      "model_0 vs model_2 \n",
      " [[ 4.  2.]\n",
      " [ 2.  2.]] \n",
      "\n",
      "model_1 vs model_2 \n",
      " [[ 5.  1.]\n",
      " [ 0.  4.]] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from mlxtend.evaluate import mcnemar_tables\n",
    "\n",
    "y_true = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])\n",
    "\n",
    "y_mod0 = np.array([0, 1, 0, 0, 0, 1, 1, 0, 0, 0])\n",
    "y_mod1 = np.array([0, 0, 1, 1, 0, 1, 1, 0, 0, 0])\n",
    "y_mod2 = np.array([0, 0, 1, 1, 0, 1, 1, 0, 1, 0])\n",
    "\n",
    "tb = mcnemar_tables(y_true, \n",
    "                    y_mod0, \n",
    "                    y_mod1,\n",
    "                    y_mod2)\n",
    "\n",
    "for key, value in tb.items():\n",
    "    print(key, '\\n', value, '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## mcnemar_tables\n",
      "\n",
      "*mcnemar_tables(y_target, *y_model_predictions)*\n",
      "\n",
      "Compute multiple 2x2 contigency tables for McNemar's\n",
      "test or Cochran's Q test.\n",
      "\n",
      "**Parameters**\n",
      "\n",
      "- `y_target` : array-like, shape=[n_samples]\n",
      "\n",
      "    True class labels as 1D NumPy array.\n",
      "\n",
      "\n",
      "- `y_model_predictions` : array-like, shape=[n_samples]\n",
      "\n",
      "    Predicted class labels for a model.\n",
      "\n",
      "**Returns**\n",
      "\n",
      "\n",
      "- `tables` : dict\n",
      "\n",
      "    Dictionary of NumPy arrays with shape=[2, 2]. Each dictionary\n",
      "    key names the two models to be compared based on the order the\n",
      "    models were passed as `*y_model_predictions`. The number of\n",
      "    dictionary entries is equal to the number of pairwise combinations\n",
      "    between the m models, i.e., \"m choose 2.\"\n",
      "\n",
      "    For example the following target array (containing the true labels)\n",
      "    and 3 models\n",
      "\n",
      "    - y_true = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])\n",
      "    - y_mod0 = np.array([0, 1, 0, 0, 0, 1, 1, 0, 0, 0])\n",
      "    - y_mod0 = np.array([0, 0, 1, 1, 0, 1, 1, 0, 0, 0])\n",
      "    - y_mod0 = np.array([0, 1, 1, 1, 0, 1, 0, 0, 0, 0])\n",
      "\n",
      "    would result in the following dictionary:\n",
      "\n",
      "\n",
      "    {'model_0 vs model_1': array([[ 4.,  1.],\n",
      "    [ 2.,  3.]]),\n",
      "    'model_0 vs model_2': array([[ 3.,  0.],\n",
      "    [ 3.,  4.]]),\n",
      "    'model_1 vs model_2': array([[ 3.,  0.],\n",
      "    [ 2.,  5.]])}\n",
      "\n",
      "    Each array is structured in the following way:\n",
      "\n",
      "    - tb[0, 0]: # of samples that both models predicted correctly\n",
      "    - tb[0, 1]: # of samples that model a got right and model b got wrong\n",
      "    - tb[1, 0]: # of samples that model b got right and model a got wrong\n",
      "    - tb[1, 1]: # of samples that both models predicted incorrectly\n",
      "\n",
      "**Examples**\n",
      "\n",
      "    For usage examples, please see\n",
      "    [http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar_tables/](http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar_tables/)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('../../api_modules/mlxtend.evaluate/mcnemar_tables.md', 'r') as f:\n",
    "    s = f.read() \n",
    "print(s)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
